Modern computing and storage systems manage increasingly large volumes of data that originates from a variety of data sources. For example, streaming data collected from a myriad of information sensing devices (e.g., mobile phones, online computers, RFID tags, sensors, etc.) and/or operational sources (e.g., point of sale systems, accounting systems, CRM systems, etc.) is growing exponentially. With the proliferation of Internet of Things (IoT) rollouts, the variety, volume, and velocity of data ingested by computing and storage systems continues to explode. Oftentimes the measurements, observations or transactions deriving from sensing devices and/or operational sources are structured as key-value pairs. The sheer number of data sources that produce such measurements, observations or transactions results in a high volume and high velocity of data to be managed by the computing and storage systems.
In some cases, a distributed storage system is implemented to manage the ever-increasing storage capacity and storage I/O demands. Distributed storage systems can aggregate various physical storage facilities (e.g., at multiple compute nodes in a cluster) to create a logical storage facility where data may be efficiently distributed according to various devices or device characteristics (e.g., storage access latency, storage cost, etc.), subdivisions (e.g., by compute node), and/or performance objectives (e.g., resource usage balancing, data durability, data availability, etc.). For example, a storage facility might have two storage tiers comprising an in-memory storage tier (e.g., a tier that is implemented using random access memory (RAM)) and an on-disk storage tier (e.g., a tier implemented using solid state storage (SSD) and/or hard disk drives (HDDs)). The in-memory storage tier provides low latency access to data in that tier (e.g., “hot” data) and the on-disk storage tier provides low cost storage of other data (e.g., “cold” data).
Unfortunately, managing increasingly larger volumes of data that originate from disparate data sources in computing and storage systems can present challenges. In some cases, ingested data (e.g., key-value pairs) might be stored in an in-memory storage tier using a certain structure, such as a binary search tree (BST), to facilitate low latency access to the values using fast access by key. Concurrently, as key-value pairs are received, a log file comprising the received key-value pairs are stored in an on-disk storage tier (e.g., to assure data retention even in the case of a service interruption).
At some moment in time, the amount of memory storage consumed by the BST in the in-memory storage tier might breach some predetermined size or utilization threshold. Or, at some moment in time, the depth or breadth of the BST might grow to an extent that reduces the search efficiency of the tree below some acceptable search efficiency threshold. Some legacy techniques address such size and efficiency issues by creating instances of a key index table and associated value table in disk storage to accept a transfer (e.g., a “flush” or a “purge”) of the then-current key-value pairs from memory storage. This flush of the data from the in-memory storage tier to an on-disk storage tier results in reclamation of storage capacity in the in-memory tier. However, in high velocity and high volume data environments (e.g., IoT environments, etc.), the computing, networking, and storage resources consumed to continually create the on-disk tables to store the purged in-memory data can become significant. What is needed is a technological solution for efficiently managing access and durability of large volumes of key-value pairs while observing characteristics and limitations of the multi-tier storage facilities.
What is needed is a technique or techniques to improve over legacy techniques and/or over other considered approaches. Some of the approaches described in this background section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.